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ABSTRACT 

This  report  describes  and  gives  the  results  of  an  experiment  designed 
to  investigate  the  psychological  soundness  of  a  proposed  unified  control 
construct,  the  KAIL  selector.   The  KAIL  selector  subsumes  several  traditional 
control  constructs  including  if-then-else ,  case,  and  while.   The  experiment 
compared  two  sets  of  control  constructs  to  determine  their  effect  on  under- 
standing.  One  set  consisted  of  the  traditional  if-then-else ,  case ,  and  while; 
the  other  set  consisted  of  a  simplified  KAIL  selector.   Results  showed  that 
subjects  understood  programs  better  when  they  were  written  using  the  KAIL 
selector. 


iii 

ACKNOWLEDGMENTS 

Professor  Wilfred  J.  Hansen  deserves  a  special  thanks  for  guiding 
me  through  this  study.   He  spent  many  hours  discussing  the  experiment  with  me 
and  suggested  numerous  modifications  and  improvements. 

I  appreciate  the  help  given  by  Professor  Richard  G.  Montanelli  on 
the  statistical  aspects  of  this  study.   He  also  read  the  manuscript  and  offered 
many  helpful  comments. 

I  appreciate  the  CS201  students  who  gave  their  time  to  take  the 
experiment . 

Many  others  helped  in  numerous  ways.   Barr  Segal  assisted  me  in 
preparing  data  for  the  statistical  package  (SOUPAC).   Dave  Eland  and  Paul 
McNabb  suggested  several  improvements  for  the  experiment.   Professor  George  H. 
Friedman  gave  technical  and  administrative  assistance,  and  Kathy  Gee  typed 
this  report. 


iv 
TABLE  OF  CONTENTS 

Page 

1.  INTRODUCTION 1 

1.1  Objective 1 

1.2  KAIL 1 

1.3  Related  Work 2 

2.  THE  KAIL  SELECTOR 3 

2.1  Basic  Selector   3 

2.2  Defaults 5 

2.3  Else 5 

2.U  Sel-tags 6 

2.5  Iteration 6 

2.6  Actions 7 

2.7  Blocks  and  Expressions   8 

3.  THE  EXPERIMENT 9 

3.1  Hypotheses 9 

3.2  Subjects 11 

3.3  Teaching  the  Languages lU 

3.1+  Problems  and  Programs 16 

3.5  Question-Answer  Session  17 

k.      RESULTS 22 

U.l  Observable  Measures  of  Understanding   22 

U.2  Difficulties 22 

U.3  Background 23 

h.k     Performance  Statistics   23 


V 

Page 

U.5  Learning  Statistics  26 

U.6  Problem  Comparison   26 

U.7  Correlations 28 

5.   CONCULSION 32 

5.1  Summary  of  Results 32 

5.2  Suggestions  for  Further  Research   32 

REFERENCES 3^ 

APPENDIX  A.   FULL  GRAMMAR  OF  THE  KAIL  SELECTOR 35 

APPENDIX  B.   SUBJECT  BACKGROUND  DATA  36 

APPENDIX  C.   PROBLEMS  AND  PROGRAM  SOLUTIONS   39 

APPENDIX  D.   QUESTIONS  kk 

APPENDIX  E.   PERFORMANCE  STATISTICS   U6 

APPENDIX  F.   CORRELATIONS 53 


vi 

LIST  OF  TABLES 
Table  Page 

1  Group  Averages  25 

2  Self-evaluations  27 

3  Average  Learning  and  Reviewing  Times  27 

k                Correlations  with  Correct  Answer  Statistics   29 

5       Data  Correlated  with  Class  Standard  Score   30 


vii 

LIST  OF  FIGURES 
Figure  Page 

1  Design  Layout   10 

2  Introduction  to  the  Experiment  12 

3  Experiment  Overview   13 

U              Instructions  18 

5  Subjective  Self-evaluation  18 

6  Preparation  for  Question-Answer  Session   19 

7  Possible  Requests  for  Help 19 

8  Typical  Question  20 


1 .   INTRODUCTION 

1.1  Objective 

Historically,  programming  languages  have  been  conceived  and 
developed  by  individuals  or  relatively  small  groups  [l].   Language  features 
have  arisen  from  the  designer's  own  experiences,  but  little  or  no  effort  has 
been  made  to  determine  if  the  design  is  psychologically  sound  [2]. 

This  report  describes  an  experiment  to  investigate  the  psycho- 
logical soundness  of  a  new  control  construct  called  the  KAIL  selector  [3]. 
The  selector  (outlined  in  chapter  2)  is  the  dominant  control  construct  of 
the  KAIL  programming  language  [k] . 

1.2  KAIL 

This  work  is  part  of  the  Automated  Computer  Science  Education 
System  (ACSES)  project  [5]  which  uses  the  PLATO  CAI  system  [6]  to  teach 
elementary  computer  science.   PLATO  runs  on  a  CDC  Cyber  73  dual  processor 
with  two  million  60-bit  words  of  extended  core  storage.  Attached  to  the 
processor  are  over  nine  hundred  plasma  display  terminals,  which  were  in- 
vented by  the  PLATO  group  for  this  project.   The  peak  comfortable  system 
load  is  about  four  hundred  simultaneous  users. 

Under  PLATO,  all  programs  are  written  in  a  special  system  langu- 
age called  TUTOR  [7]  which  mixes  FORTRAN-like  flow  of  control  with  peculiar 
CAI  sequencing  and  sophisticated  answer  judging  facilities.   The  language 
KAIL  is  an  attempt  to  influence  the  development  of  TUTOR  by  showing  that 
modern  control  constructs  can  be  combined  with  the  higher  level  features 
of  TUTOR. 


1.3  Related  Work 

In  1971 »  Gerald  M.  Weinberg  wrote  The  Psychology  of  Computer  Pro- 
gramming [8].   Its  stated  purpose  was  "to  trigger  the  beginning  of  a  new 
field  of  study:   computer  programming  as  a  human  activity."  In  the  book,  he 
advocated  the  use  of  experimentation  as  a  method  to  study  programming  and 
suggested  that  human  factors  be  considered  in  language  design. 

Since  the  appearance  of  Weinberg's  book,  several  researchers  have 
conducted  psychological  experiments  on  programming  language  features.   Using 
inexperienced  programmers,  three  psychologists  concluded  that  nested  if 
programs  are  easier  to  understand  than  corresponding  goto  programs  [9].   Larry 
Weissman  listed  and  catagorized  factors  affecting  the  complexity  of  programs, 
began  development  of  a  methodology  for  studying  the  psychological  complexity 
of  computer  programs,  and  conducted  some  experiments  attempting  to  determine 
which  factors  or  combinations  of  factors  reduce  complexity  [10,  11].  At  the 
University  of  Indiana,  memorization  tests  were  used  to  measure  the  effect  of 
program  structure  on  understanding  [12].   Recently,  John  D.  Gannon  completed 
a  Ph.D.  thesis  for  which  he  gathered  empirical  evidence  to  support  or 
discredit  specific  language  design  decisions  [13]. 

Like  Gannon's  work,  the  KAIL  selector  experiment  was  an  approach  to 
language  design.   It  examined,  however,  a  new  syntatic  and  semantic  construct 
rather  than  a  controversy  between  existing  features.   The  methodology  of 
this  experiment  resembled  Weissman' s.   One  major  difference  was  the  use  of 
automated  rather  than  manual  techniques.   Because  the  PLATO  CAI  system 
administered  the  experiment  on-line,  the  language  constructs  could  be  taught 
as  part  of  the  experiment;  and  precise  timing  data  and  all  other  relevant 
information  could  be  gathered. 


2.   THE  KAIL  SELECTOR 

2.1  Basic  Selector 

The  KAIL  selector  differs  from  most  typical  high  level  control 
constructs  in  three  ways: 

1.  It  combines  flow  of  control  with  "answer  judging," 
the  process  by  which  a  Computer-Aided  Instruction 
(CAI)  program  determines  whether  a  student  reply 
matches  one  of  the  replies  anticipated  by  the  lesson 
author. 

2.  It  combines  most  of  the  other  control  constructs  that 
have  been  widely  used,  e.g.,  if-then-else,  case ,  cond, 
while,  ... 

3.  The  syntax  suggests  an  answer  to  the  trivial,  yet  thorny, 
question  of  how  to  end  a  nested  block  of  code.   Previous 
proposals  include:   end  of  one  statement,  end,  fi-esac- 
elihw  . . . ,  endif-endcase-endwhile  ... 

This  KAIL  selector  is  the  result  of  a  search  for  a  syntactic  struc- 
ture for  CAI  answer  judging.   In  answer  judging  the  common  control  operation 
is  to  compare  a  student  response  against  a  set  of  expected  correct  and 
incorrect  responses.   This  suggests  a  control  syntax  where  a  given  expression 
(usually  the  student's  reply)  is  tested  against  a  number  of  alternatives. 
The  full  selector  syntax  is  given  in  appendix  A,  but  will  be  introduced 
piecemeal  in  the  sections  below. 

The  essential  selector  syntax  can  be  sketched  as 

selector  :  "["   statement -sequence  control  choices  "]" 

control   :  "if"  expression 


choices   :  X 

"|"  relation  expression  ":"   statement-sequence   choices 

where  expression,  relation,  and  statement-sequence  are  defined  by  the  base 
language.  (The  two  expressions  are  referred  to  as  control-expression  and  choice- 
expression,  respectively.)  The  selector  is  executed  in  four  steps: 

1.  Execute  the  statement-sequence  in  the  selector. 

2.  Evaluate  the  control-expression  and  save  its  value  in 
a  temporary,  t. 

3.  Test  each  choice  in  turn  until  a  "selected  choice"  is 
found  such  that 

(t  relation  choice-expression)  yields  true. 
k.      Execute  the  statement-sequence  in  the  selected  choice. 
If  no  selected  choice  is  found,  step  four  is  omitted. 

This  simple  example  tests  a  student's  arithmetic  abilities: 

x  :=  rand(25);  y  :=  rand(25);  comment  set  x  and  y  to  random  integers 

in  [1,25]; 
write  What  is  <(x)>  +   <(y)>?  ; 
[accept  reply;  if_  reply 

=  x+y:   write  Very  good;  right_count  :=  right_count  +  1; 

=  x*y:  write  Add,  don't  multiply; 

>  x+y+20:   tx  :=  x  div  10;  ty  :=  y  div  10; 

write  But  <(tx+l)>0  +  <(ty+l)>0  is  only  <(tx+ty+2)>0! ; 

]; 


2.2  Defaults 

The  key  to  the  adaptability  of  the  KAIL  selector  is  its  default 
values.   The  expressions  and  relations  may  be  omitted  with  the  following  result, 

control-expression  true 

relation  = 

choice-expression   (l)  true 

choice-expression   (2)  false 

choice-expression   (i)  i-2     (i  >_  3) 

(in  TUTOR,  -1  is  true,  and  0  is  false.   In  a  spirit  of  compatibility,  KAIL 
has  adopted  these  conventions.)   When  the  choice-expression  is  omitted,  the 
relation  and  ":"  must  also  be  left  out. 


By  proper  choice  of  default  values,  the  selector  has  the  semantics 
of  several  other  control  constructs,  including:  if-then-else ,  labelled-case 
(similar  to  LISP's  SELECT),  unlabelled-case,  and  COND.   For  example,  for 
if-then-else  there  are  only  two  choices  and  no  choice-expressions  (e.g., 
[if  x  >_  y  |  max  :=x  |  max  :=y]).   For  a  labelled-case ,  the  relations  are 
omitted  and  the  choice-expressions  are  constants.   (it  would  be  closer  to 
a  labelled-case  if  the  quotes  could  be  omitted  from  string  constants,  but 
then  the  selector  could  not  be  mapped  into  such  a  variety  of  constructs.) 
2.3  Else 

One  additional  choice  is 

choice   :   "  |  else  :  "  statement-sequence 
If  evaluation  reaches  this  choice,  it  is  always  the  "selected  choice"  and 
its  statement-sequence  is  executed.   This  corresponds  to  the  out  clause  in 
the  ALGOL  68  case  construct.   Notice  the  distinction  between  "|"  and 
"|  else:"  in  an  if-then-else :   without  the  else,  the  second  choice  will 
only  be  selected  if  the  control-expression  yields  false;  with  the  else, 
any  value  but  true  will  select  the  second  choice. 


2.k      Sel-tags 

Selectors  can  become  deeply  nested.   To  help  the  reader  and  compiler 
match  brackets,  the  programmer  may  tag  them  with  matching  sel-tags: 

selector   :   sel-tag  "_"   selector  "_"   sel-tag 
where  the  two  sel-tags  are  both  the  same  unique  identifier.   In  practice,  the 
separation  of  the  tag  from  the  bracket  by  the  unobtrusive  underline  has  a 
reasonably  good  appearance.   (intuitively,  this  is  so  because  bracketed  expres- 
sions are  often  surrounded  by  spaces.) 
2. 5  Iteration 

The  KAIL  selector  unifies  iteration  and  selection, 
selector   :   ".*["   statement-sequence  [loop-control  choices]  "]*" 
loop-control   :   "while"  expression  j  "until"  expression 
For  iteration,  a  fifth  step  is  added  to  the  selector  semantics: 
5.   For  while ,  if  no  "selected  choice"  is  found,  exit; 
otherwise,  return  to  step  one.   For  until,  if  no 
"selected  choice"  is  found,  return  to  step  one; 
otherwise,  exit. 
Notice  that  "*"  distinguishes  iteration  from  selection  both  at  the  beginning 
and  ending  of  a  loop,  and  that  the  keywords,  while  and  until  reinforce  this 
notion  from  within  the  construct. 

Just  as  the  basic  selector  maps  into  several  branching  constructs, 
the  iteration  construct  maps  into  several  existing  loop  constructs,  including: 
while,  repeat-until ,  loop-while-repeat  [lU],  and  the  nesting  of  a  single  case 
statement  inside  of  a  while.   For  example,  for  while ,  there  is  only  one  choice, 
and  either  the  control-expression,  the  choice-expression,  or  an  appropriate 
combination  serves  as  the  predicate.   For  repeat-until,  the  body  of  the  loop 
is  the  initial  statement -sequence,  and  no  choice  appears. 


As  an  example,  consider  the  inner  loop  of  Hoare's  "Quicksort" 
algorithm  [15]. 

comment  m  <  n  and  A(m-l)  exists  and  A(m-l)  <  A(i)  for  m  _^  i  _<  n; 
i  :=  m;   j  :=  n-1;   v  :=  A(n) ; 

#[    *[  while  |  A(i)  <  v  :  i  :=  i+1;  ]*;  comment  incr  i; 
*[  vhile  |  A(j)  >  v  :  j  :=  j-1;  ]*;  comment  deer  J; 
vhile  1  <  J 

A(i)  :=:  A( j ) ;   comment  svap  A(i)  and  A(j); 

]•; 

This  example  illustrates  hov  the  selector  solves  the  problem  of 
looping  "n  and  a  half  times"  [lU]  vithout  introducing  an  exit  or  a  goto. 
Such  problems  arise  often  in  practive,  and  sometimes  almost  demand  a  goto. 
Goto-free  programmers  usually  resort  to  code  duplication  and/or  redundant 
tests. 
2.6  Actions 

Actions  alter  normal  flov  of  control  patterns. 

choice   :   "I"  [relation  expression  ":"]  statement -sequence  [action] 
action   :   "exit"  [sel-tag]    "again"  [sel-tag] 
Exit  causes  control  to  leave  the  selector;  again  directs  control  to  the  begin- 
ning of  the  initial  statement-sequence.   Sel-tags  allow  exit  and  again  from 
within  nested  selectors. 

Actions  may  be  used  in  the  CAI  example  of  section  2.1  to  force  the 
student  to  answer  correctly  before  proceeding.   A  programmer  may  use  selection 
and  again ,  but  iteration  and  exit  would  serve  just  as  well: 


x  :=  rand(25);  y  :-  rand(25); 
vrite  What  is  <(x)>  +  <(y)>?  . 
[  accept  reply;   if  reply 

x+y  :  vrite  Very  good;   right_count  :=  right_count+l; 
x*y  :  vrite  Add,  don't  multiply;  again; 
j  >x+y+20  :  tx  :=  x  div  10;  ty  :=  y  div  10; 

vrite  But  <(x+l)>0  +  <(ty+l)>0  is  only  <(tx+ty+2)>0! ; 
again; 

]; 

2.7  Blocks  and  Expressions 

If  both  the  control  and  the  choice-sequences  are  omitted,  the 
selector  groups  statements  into  a  block. 

The  selector  may  be  used  in  an  expression  context  by  replacing 
the  statement-sequence  in  every  choice  vith  an  expression.  There  must  be  at 
least  one  choice.   Note  that  this  construct  permits  statements  inside 
expressions . 


3.   THE  EXPERIMENT 

3.1  Hypotheses 

Observations  on  the  characteristics  of  the  KAIL-selector  and  its 
use  in  programming  lead  to  the  hypothesis  that  the  KAIL  selector  is  easier  to 
understand  than  an  equivalent  set  of  traditional  constructs.   The  selector 
unifies  selection  and  iteration  and  subsumes  CAI  answer  judging  and  most 
typical  high  level  control  constructs.   This  should  enhance  understanding 
because  programmers  would  only  need  to  know  one  control  construct,  not  several, 
In  the  selector  snytax,  symbols  replace  keywords  in  some  instances.   This 
should  also  enhance  programmer  understanding  because  it  tightens  the  code 
and  packs  more  information  into  a  program  listing.   For  instance,  the  closure 
itself  indicates  whether  or  not  the  statement  is  to  be  repeated. 

In  order  to  test  the  hypothesis,  it  is  first  necessary  to  devise  a 
method  for  measuring  understanding.   A  subject's  ability  to  comprehend  is  a 
function  of  several  factors.   These  include  the  following: 

Environmental  factors: 

1.  Educational  experience 

2.  Knowledge  of  programming  languages 

3.  Basic  intelligence 
h.      Motivation 

Observable  factors: 

1.  Speed  at  answering  questions 

2.  Accuracy  at  answering  questions 

3.  Self-evaluation  of  understanding 

Together,  these  factors  comprise  a  model  of  subject  understanding. 

This  model  of  understanding  allows  the  hypothesis  to  be  broken 
down  into  measurable  variables.   The  effects  of  environmental  factors  can  be 
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minimized  by  grouping  subjects  so  that  among  groups,  background  and  motivation 
are  similar.   The  hypothesis  can  then  be  tested  by  measuring  the  observable 
factors.   If  the  hypothesis  is  correct,  subjects  working  with  the  selector 
should  be  able  to  answer  more  questions  and  answer  them  quicker;  they  should 
make  fewer  mistakes;  and  they  should  give  higher  self-evaluations. 

To  test  these  hypotheses  an  experiment  was  devised.   Two  versions  of 
two  programs  were  written:   one  version  in  KAIL,  one  in  a  language  with  standard 
control  constructs.   Each  subject  was  assigned  to  one  of  four  groups  as  shown 
in  figure  1. 


language 
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u 
o 


bO 
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U 
ft 


#1,  #2 


#2,  #1 


group  #1 

group  #2 

group  #3 

group  #U 

Figure  1.   Design  Layout 
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3.2  Subjects 

The  subjects  were  volunteers  from  CS  201,  Spring  Semester,  1975- 
Most  were  sophomores  or  juniors  and  had  previously  taken  one  programming  course 
using  FORTRAN,  PL/1,  or  both.   The  typical  subject's  programming  ability  con- 
sisted of  experience  in  one  or  two  languages  and  some  knowledge  of  a  smattering 
of  others.  About  half  had  previously  used  PLATO,  some  quite  extensively. 
The  experiment  gathered  this  background  information  along  with  each  subject's 
name  and  social  security  number  which  was  later  used  to  associate  a  student's 
CS  201  grade  with  the  data.   Appendix  B  contains  a  table  which  summarizes 
this  information. 

To  motivate  subject  participation,  it  was  explained  to  a  general 
CS  201  class  session  that  an  experiment  was  being  conducted  on  PLATO  to  deter- 
mine why  seme  programming  languages  are  easier  to  understand  than  others,  and 
it  was  pointed  out  that  this  would  be  an  opportunity  to  become  acquainted  with 
the  PLATO  system  and  participate  in  an  unusual  experiment.  Fifty-three  percent 
of  the  CS  201  students  eventually  responded  and  came  to  one  of  several  sessions 
when  terminals  were  reserved. 

In  each  session  as  the  actual  experiment  began,  this  announcement 
was  made,  "You  will  not  be  graded  on  how  well  you  do;  however,  please  treat 
this  experiment  like  an  exam."  After  subjects  signed  in,  the  system  welcomed 
them  and  encouraged  them  to  do  their  best  as  illustrated  in  Figure  2. 

If  a  subject  was  not  acquainted  with  PLATO,  the  system  taught  the 
use  of  function  keys  and  other  basics.   (Since  the  experiment  only  required 
short  responses,  often  only  a  single  keypress,  extensive  PLATO  experience 
was  unnecessary. ) 
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Exper  1  ment    Overv  1  euj 


I .   Background  i  n format  1  on 


1 1 .   Programm  i  ng  1  anguage  descry  pt  i  on 
ft .   Dec  1  ara  b  i  oris 

Rss i gnment  st  at  ement s 

Input -output 
Se  1  ect  i  on 
Iter at  ion 
Express i ons 
Samp  I e  program 


TTT 


Problem  1 

ft .      Exp  1 anat i on  o f  prob 1 em 

B .  I n i t i t i a  1    st udy  o f   program 

C .  Se 1 f   eva 1 uat i on    1 

D .  Qu i z 

E .  be  1 f   eva 1 uat l on    2 


IV.       Problem    2 


same  as  above 


press   Ihextj 


Figure  3.      Experiment  Overview 
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The  system  then  outlined  the  experiment  for  all  subjects  (Figure  3). 

After  subjects  completed  the  background  information  section,  the 
experiment  assigned  each  to  one  of  the  four  cells  by  first  computing  a  level 
number 

level  number  =  f( education  level,  programming  experience) 
and  then  assigning  subjects  on  the  same  level  successively  to  cell  1,  2,  3, 
h,   1,  2,  ...  ,  etc.   (As  it  turned  out,  f  was  unnecessarily  complicated 
because  the  students  in  CS  201  were  already  reasonably  similar.   Even  worse, 
f  divided  the  subjects  unevenly  among  the  four  cells.   Successively  assigning 
subjects  to  cells  would  have  been  sufficient.) 
3.3  Teaching  the  Languages 

The  two  objectives  of  the  instruction  section  of  the  experiment 
were  to  teach  the  syntax  and  semantics  of  the  assigned  language  and  to  establish 
some  minimum  level  of  understanding. 

Declarations,  assignment  statements,  and  input-output  statements 
were  identical  in  both  languages. 

declaration   :   integer  identifier-list     character  identifier-list 


assignment-statement   :   variable  <=  expression 

input -output -statement   :   read  identifier -list     write  identifier-list 

identifier-list   :   identifier     identifier,  identifier-list 

Selection,  iteration,  and  expressions  differed  in  the  two  languages. 
The  selector  syntax  chosen  for  use  in  the  KAIL-like  language  emphasized  the 
basic  selector  characteristics.   The  other  language  was  akin  to  ALGOL  68* 
Keywords  were  used  liberally,  and  selection  and  iteration  were  different 
constructs.   Also,  using  the  ALGOL  68  closures  fi,  esac ,  and  od  made  the 
introduction  of  begin  and  end  unnecessary. 
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In  the  ALGOL-like  language  selection  had  two  forms: 
selection   :   if-statement     case-statement 
if-statement   :   "if"  integer-expression  "then"  statement-sequence 

"else"   statement -sequence  "f i" 
case-statement   :   "case"   integer-expression  choices   "esac" 
choices   :   ":"   [  subscript  ]   statement-sequence 

":"   [  subscript  ]  statement-sequence  choices 

For  the  if-statement,  if  the  integer-expression  evaluated  to  0,  the  then 
statement-sequence  was  executed;  if  it  evaluated  to  1,  the  else  statement- 
sequence  was  executed;  otherwise,  neither  statement-sequence  was  executed, 
(in  both  languages,  boolean  expressions  evaluated  to  0  if  true  and  1  if 
false. )   For  the  case-statement,  the  integer-expression  was  evaluated  and 
saved  in  a  temproary  t;  then  the  statement -sequence  in  the  t+lst  choice  was 
executed.   If  t+1  was  out  of  range,  no  statement-sequence  was  executed.   For 
example , 

case  k  :8  write  "a";  ::  write  "b";  :2<  write  "c";  esac; 

output  "a"  if  k  is  0,  "b"  if  it  is  1,  "c"  if  it  is  2, and  nothing  if  k  is 
anything  else. 

In  the  KAIL-like  language, selection  had  one  form: 
selection   :   "["  integer-expression  choices  "]" 
choices   :   "|"   [  subscript  ]   statement-sequence  choices 
When  executed,  the  integer-expression  was  evaluated  and  saved  in  a  temporary  t 
then  the  statement-sequence  in  the  t+lst  choice  was  executed.   If  t+1  was  out 
of  range,  no  statement -sequence  was  executed.   In  this  language,  iteration 
appeared  in  exactly  the  same  form  as  selection  except  that  "*"  was  placed  in 
front  of  the  left  bracket  and  after  the  right  bracket  to  indicate  repetition. 

iteration   :   *  selection  * 
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The  semantics  of  iteration  were  identical  to  the  semantics  of  selection  except 
that  as  long  as  t+1  was  in  range,  the  statement  was  repeatedly  executed.   For 
example , 

sum    <>   J0f ;        1     .>    1  ; 
L    i-n    1    sum   <t»   sum    +    i;       i    <r    i    +    1 ;     ]*; 

summed  the  digits  from  1  to  n. 

In  the  ALGOL-like  language,  iteration  had  the  form: 
iteration   :   "while"   integer-expression  "do"   statement -sequence   "od" 
The  statement-sequence  was  repeatedly  executed  as  long  as  the  integer-expression 
was  true  (had  value  0). 

In  both  languages,  simple  expressions  were  identical,  hut  selection 
expressions  were  different.   These  had  the  same  form  as  selection  statements 
only  the  statement -sequences  were  replaced  by  expressions. 

To  establish  some  minimum  level  of  understanding,  the  experiment 
asked  each  subject  six  questions  as  it  taught  the  language.   Each  question 
tested  some  aspect  of  the  language;  moreover,  the  questions  served  to 
familiarize  subjects  with  the  type  of  question-answer  interaction  required 
later  in  the  experiment. 
3.h     Problems  and  Programs 

The  two  problems  were  chosen  for  the  experiment  such  that  the  pro- 
grams would  exemplify  all  important  variations  of  the  language  constructs. 
One  problem  was  to  find  all  automorphic  numbers  less  than  or  equal  to  a  number 
read  at  the  beginning  of  the  program.   The  solution  to  this  problem  read 
naturally  in  the  ALGOL-like  language  because  it  made  common  use  of  if -then-else, 
while,  and  case.   The  other  program  analyzed  arithmetic  expressions  involving 
the  symbols  "+",  "-" ,  and  "a".   The  solution  to  this  problem  took  advantage  of 
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the  selector  by  eliminating  a  level  of  syntactic  structure.   Appendix  C  gives 
the  description  of  these  problems  and  the  four  programs  used  in  the  experiment. 

It  is  important  to  point  out  that  one  program  was  designed  to  favor 
the  ALGOL-like  language  group  and  the  other  to  favor  the  KAIL-like  language 
group.   This  leads  to  a  further  hypothesis:   the  group  using  the  KAIL-like 
language  taking  the  arithmetic  expression  problem  last  should  perform  best. 
This  is  because  they  would  have  the  advantage  of  working  the  program  which 
maximizes  the  used  of  the  selector  after  becoming  familiar  with  the  KAIL-like 
language. 
3 .5  Quest ion -Answer  Session 

After  describing  one  of  the  problems,  the  experiment  proceeded  with 
the  instructions  shown  in  Figure  h. 

A  subject  studied  the  program  for  100  seconds  and  then  made  a  sub- 
jective self-evaluation  as  shown  in  Figure  5. 

The  experiment  then  prepared  the  subject  for  the  question-answer 
session  and  encouraged  good  performance  (Figure  6). 

All  subjects  answered  identical  questions.   The  10  questions  for 
each  problem  appear  in  appendix  D.   Before  each  question,  the  subject  could 
review  by  pressing  "help",  "shift-help",  or  "data"  as  shown  in  Figure  J. 
On  pressing  the  helpkey,  a  brief  six-line  explanation  appeared  on  the  bottom 
of  the  page;  the  program  remained  above.   By  pressing  "shift-help",  the 
subject  gained  access  to  the  instructional  material  used  in  teaching  the 
language.   On  pressing  "data",  the  problem  explanation  reappeared. 

When  the  subject  pressed  "shift-next",  Figure  8  shows  how  the  system 
presented  the  next  question  along  with  a  reminder  to  work  quickly. 

The  subject  either  answered  the  question  correctly  or  gave  up  after 
2  minutes,  or  3,  or  3  1/2,   or  3  3A,  etc.,  up  to  h   minutes.   The  system 
noted  the  elapsed  time  for  each  input  and  recorded  all  wrong  answers. 
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When  you  press  SSSBIIhextj,  the  program^  will  appear.   Fifter 
100  seconds,  it  will  disappear;  and  you.  will  be  asked  to 
evaluate  yourself  on  how  well  you.  think  you  understand  it. 


T1 


"his  is  not  the  last  time  you  will  see  the  program, 
so  your  only  task  at  present  is  to  understand  how  the 
the  program  works.   For  your  convenience,  you  will  be  told 
how  much  time* is  left  about  every  10  seconds. 


Figure  k.      Instructions 


Enter  the  number  which  best  describes  your 
feelings  about  how  well  you  understand  the  program 
at  t  h i s  1 1 me . 


0.   I  don't  understand  it  at  all. 


I  get  the  gist  of  how  it  works 


I  can  explain  How  it  works. 


I  understand  it  perfectly. 


:e  numbers  which  are  not  labeled  ■■r-r-'e   meant  t- 
iations  of  understanding  in  between  those 
ire  labeled. 


Figure  5.   Subjective  Self -evaluation 
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Quiz  time1   There  are  lJ0  questions.   They  uu  1  1  appear 

one  at  a  time  on  the  same  page  as  the  program.   Before  each 

lestion,  you  will  have  the  chance  to  review  the  language 
and  the  problem  description;  but  while  you  are  answering  a 
question,  no  neJLp.  will  be  available. 


The  objective  is  to  answer  the  question  as  quickly  as 
ible.   Filter  2  minutes,  you  will  be  given  the 

tunity  to  give  up  or  continue  on  for  another  minute  or 
s:  .   Answer  the  question  to  the  to  the  best  of  your  ability 
because  your  effort  a.  will  help  us  learn  about  the  psychology 
o!  pr ogramrr  :ng  . 

Figure  6.   Preparation  for  Question-Answer  Session 


:j  i  c k    rev  1  ew  y^N^inzLPi     ext ens  i  ve   rev  1  ew 


IJDftTftj     to   see   the   program  explanation   again 
S-EvmHEXT]      for   a   question   about    this   program 

Note:       You   will    not   be   able   to-   request 
help   while   answer  ing   questions. 

Figure  T«   Possible  Requests  for  Help 
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i  nteger    i  ,  j  ,K.fn,  1 sqr ; 

read   n ; 

i    ♦    1 ;       k   ■?=   .9' ; 

wh lie    i    ^    n   do 

i  sqr   <f    i  >:  l  ; 

j    *    1 ; 

wh  lie    j  '  i    i    do    j    <f    j  *  1  .y ;    od ; 

if    i=(isqr    -    j *  (isqr*j) )    then   write    i,isqr;     fi; 


}.; 

f\ 

1 

1 

+ 

A  • 

1. 

1    . 
1   n 

1 

I 

'  I- 

1 

T 

i ; 

v 

a. 
V 

■*"■    ? 

i 

J. 

c  . 

T.- 

ff: 

List    the   values   that    k   can   have   during   program 
execution.        (Separate   values   with    ","    or   space.) 


Figure   8.      Typical  Question 
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After  completing  question  10,  the  subject  made  a  final  self- 
evaluation  on  the  0-9  scale  as  before.  The  second  problem  was  then  presented 
in  the  same  fashion  as  the  first,  and  the  experiment  was  complete. 
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k .   RESULTS 

k .1     Observable  Measures  of  Understanding 

As  explained  in  section  3.1,  the  effect  on  understanding  was  to  be 
measured  in  several  ways: 

1.  number  of  questions  answered  correctly 

2.  number  of  questions  answered  correctly  on  first  attempt 

3.  average  time  taken  to  answer  a  question  correctly 
k.      initial  self-evaluation 

5.  final  self -evaluation 

6.  difference  between  final  and  initial  self-evaluation 

These  performance  statistics  are  summarized  in  appendix  E,  and  the  important 
results  are  discussed  below. 

The  experiment  also  gathered  statistics  on  the  amount  of  time  each 
subject  spent  in  learning  and  in  reviewing  the  assigned  language  and  in 
studying  the  problem  descriptions.   These  results  are  mentioned  in  the  text 
below.   Correlations  among  performance  data,  language  knowledge,  CS  201  grade, 
and  the  amount  of  time  spent  learning  the  language  are  found  in  appendix  F 
and  discussed  below. 
U.2  Difficulties 

Several  difficulties  were  encountered  during  the  experiment.   Some 
of  these  were  expected;  others  were  unexpected. 

Since  the  PLATO  CAI  system  can  only  judge  answers  correctly  if  they 
are  anticipated  by  a  lesson  author  (e.g.,  10000  and  1,000  are  different),  the 
experiment  recorded  all  answers  to  every  question,  whether  right  or  wrong,  and 
logged  the  appropriate  elapsed  time.  As  it  turned  out,  few  subjects  entered 
responses  which  were  judged  incorrectly.   Typing  in  "_"  (underline)  for  "-" 
(minus)  was  the  most  prevalent  of  these  few  errors.   In  these  cases,  the  raw 


23 


data  was  updated  to  reflect  the  appropriate  system  logging  as  if  these  answers 
had  been  anticipated. 

Partial  data  from  two  subjects  was  lost  because  of  a  buffer  overflow 
in  the  monitoring  program.   This  bug  was  discovered  and  fixed  before  most  of 
the  subjects  took  the  experiment. 

During  one  session,  the  system  crashed.  Of  the  sixteen  subjects  in 
this  session,  five  completed  the  experiment  before  the  crash,  nine  completed 
one  of  the  two  problems,  and  two  had  not  yet  completed  the  first  problem.  Of 
the  nine  who  completed  only  one  problem,  four  were  using  the  KAIL-like  language 
and  five  were  using  the  ALGOL-like  language.   In  addition  to  the  system  crash, 
at  another  session,  two  subjects  encountered  hardware  screen-overwrite  problems 
with  their  terminals  causing  some  of  their  data  to  be  declared  invalid.  Mostly, 
this  partial  data  was  ignored,  but  it  was  included  in  the  analysis  when 
theoretically  appropriate. 
h.3     Background 

No  significant  differences  appeared  in  any  background  data.   Appendix  B 
gives  the  background  of  the  average  subject  in  each  cell  for  all  factors  con- 
sidered.  These  statistics  are  uninteresting  if  considered  by  themselves,  but 
they  lend  credence  to  the  results  which  follow  because  they  show  that  the 
experiment  was  unbiased  with  regard  to  background. 
1+.1+  Performance  Statistics 

A  three-way  analysis  of  variance  using  the  approximate  method  of 
unweighted  means  was  used  to  compare  performance  statistics.   The  three 
factors  were  language,  order,  and  problem.   Subjects  took  both  problems, 
but  only  in  one  order  using  one  language.  These  statistics  are  found  in 
appendix  E. 
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Subjects  using  the  KAIL-like  language  on  the  average  answered  l6.7 
of  the  20  questions  correctly;  subjects  using  the  ALGOL-like  language  answered 
only  15.1  correctly.   This  result  is  significant  at  the  2.5  percent  level.   A 
chi-squared  test  validated  the  normality  assumptions.   This  result  supports 
the  hypothesis.   The  selector  construct  enabled  subjects  to  answere  more 
questions  correctly. 

Table  1  shows  how  each  of  the  four  groups  performed  on  the  number 
answered  correctly.   Group  #2,  using  the  KAIL-like  language  taking  the  auto- 
morphic  numbers  problem  first  answered  on  the  average  of  two  more  questions 
correctly  than  any  other  group.   This  result  supports  the  hypotheses  explained 
in  section  3.h.      Subjects  in  this  group  took  the  arithmetic  expression  problem 
after  having  taken  the  automorphic  numbers  problem;  and  thus,  had  the  advantage 
of  working  the  program  which  maximized  the  use  of  the  selector  after  they  had 
become  familiar  with  the  KAIL-like  language. 

A  typical  subject  using  the  KAIL-like  language  answered  12.3  of  the 
20  questions  correctly  on  the  first  try;  subjects  using  the  ALGOL-like 
language  answered  11. 7.   This  is  not  significant.   (One  might  expect  signifi- 
cance because  appendix  F  shows  a  high  correlation  between  number  correct  and 
number  initially  correct,  but  these  means  are  too  close.)   It  was  hypothesized 
(section  3-M  that  group  #2  using  the  KAIL-like  language  taking  the  automorphic 
numbers  problem  first  would  perform  best.   Table  1  shows  that  they  answered 
about  two  more  questions  correctly  on  the  first  try  than  any  other  group. 

An  analysis  of  the  average  time  taken  to  answer  a  question  correctly 
showed  no  significant  difference  between  languages.   To  obtain  these  statistics, 
raw  data  was  first  transformed  to  a  normal  distribution  with  mean  0  and  standard 
deviation  1,  and  an  average  for  each  subject  on  each  problem  was  computed. 
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Group 


Number  in  group 

Number  answered  correctly 
(20  possible) 


n_ 

19 
15-2 


Number  answered  correctly 

on  the  first  try  (20  possible)   11. T 


Standardized  mean  time  to 
answer  correctly 


#2 


0.055 


13 


17.5 


13.3 


-0.12 


13 

15 

15.0 
11.8 
O.lU 


ttk 


11 


15.8 

11.  h 
0.057 


language 


problem  order 


group  #1 
group  #2 
group  #3 
group  §h 


ALGOL-like  Automorphic  Numbers  problem  first 

KAIL-like  Automorphic  Numbers  problem  first 

ALGOL-like  Arithmetic  Expression  problem  first 

KAIL-like  Arithmetic  Expression  problem  first 


Table  1.   Group  Averages 
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This  removed  the  effects  introduced  "by  differing  degrees  of  difficulty  in  the 
questions.   Data  on  questions  answered  incorrectly  was  ignored.   The 
standardized  mean  time  to  answer  a  question  correctly  was  0.097  for  the  ALG0L- 
like  language  subjects  and  -0.033  for  the  KAIL-like  language  subjects.   Note 
that  the  lower  number,  even  though  not  significant,  means  the  subjects  using 
the  KAIL-like  language  took  less  time  and  therefore  did  better.   Again,  the 
data  in  Table  1  confirms  the  hypothesis  explained  in  section  3.^. 

Table  2  shows  how  the  average  subject  in  each  of  the  four  groups 
responded  to  the  initial  and  final  self -evaluations.   With  regard  to  language 
effects,  only  one  siginificant  result  emerged.   Disregarding  order,  the 
KAIL-like  language  subjects  thought  they  initially  understood  the  arithmetic 
expression  program  at  level  3.8  whereas  the  ALGOL-like  language  subjects  only 
thought  they  understood  at  level  2.7«   This  is  significant  at  the  2.5  percent 
level.   Across  the  entire  experiment,  trends  in  both  the  initial  and  final 
subjective  self-evaluations  favored  the  KAIL-like  language;  however,  no 
trend  was  detected  when  considering  the  difference  between  these  evaluations. 
h. 5  Learning  Statistics 

The  experiment  measured  the  time  subjects  spent  learning  and  reviewing 
their  assigned  language.   These  times  are  summarized  in  Table  3.   No  results 
were  significant.   It  is  interesting,  however,  to  note  that  the  ALGOL-like 
language   subjects  spent  on  the  average  of  9^  seconds  longer  in  learning. 
This  supports  the  hypothesis  that  the  syntax  and  semantics  of  selector  construct 
are  at  least  as  easy  to  learn  as  the  snytax  and  semantics  of  the  traditional 
constructs. 
h  .6   Problem  Comparison 

Subjects  performed  better  on  the  arithmetic  expression  problem  than 
on  the  automorphic  numbers  problem.   As  shown  in  appendix  E,  every  performance 
measure  except  average  time  to  obtain  a  correct  answer  exhibited  significant 
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Number  in  group 


Automorphic 

Numbers 

Ar 

ithmetic 

Expression 

initial 

final 

initial 

final 

2.9 

U.l 

2.9 

5.U 

2.3 

3.6 

3.U 

5.7 

2.0 

3.6 

2.5 

U.5 

2.3 

3.2 

U.2 

5.7 

group 

#1 

19 

group 

#2 

13 

group 

n 

15 

group 

#u 

11 

Zero  represents  no  understanding;  nine  represents  perfect  understanding. 
The  groups  are  the  same  as  in  Table  1. 


Table  2.   Self -evaluations 


language 

ALGOL-like 

KAIL-like 


number  in  group 
3k 
2k 


seconds  learning   seconds  reviewing 
928  20 

83U  21 


Table  3.   Average  Learning  and  Reviewing  Times 
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results.   Subjects  answered  more  questions  correctly  (p<.0l),  answered  more 
questions  correctly  on  the  first  try  (p<.02U),  thought  the  program  was  easier 
to  understand  both  before  (p<.0l)  and  after  (p<.0l)  working  on  the  questions, 
and  indicated  that  they  though  there  was  a  greater  increase  in  understanding 

(p<.025). 

k. T  Correlations 

The  correlation  data  supports  the  model  of  subject  understanding 
presented  in  section  3.1.   Table  h   shows  a  high  correlation  among  subjects  on 
the  number  of  questions  answered  correctly,  the  number  answered  correctly  on  the 
the  first  try,  and  the  time  taken  to  answer  a  question  correctly.   In  answering 
questions,  the  model  of  understanding  expected  subjects  who  understood  well 
to  perform  well  and  those  who  understood  poorly  to  perform  poorly.   Although 
self-evaluations  are  fraught  with  difficulties  because  of  their  subjective 
nature,  no  unusual  correlations  appeared.   (See  appendix  F. )   This  also 
supports  the  model  of  subject  understanding. 

Between  language  groups,  correlation  statistics  compared  reasonably 
well  except  in  the  area  of  CS  201  grades.   Table  5  shows  that  better  students 
(measured  by  class  standard  score)  using  the  KAIL-like  language  answered 
more  questions  correctly  than  the  poorer  students,  answered  quicker  especially 
on  the  automorphic  numbers  problem,  gave  higher  initial  self-evaluations,  and 
spent  less  time  learning.   Better  students  using  the  ALGOL-like  language 
answered  about  the  same  number  correctly  as  the  poorer  students ,  answered 
slightly  slower,  gave  slightly  lower  initial  self -evaluations ,  and  spent 
about  the  same  amount  of  time  learning  the  language.   It  is  somewhat  difficult 
to  interpret  these  results  in  terms  of  the  value  of  the  KAIL  selector  versus 
the  traditional  constructs.  It  is,  however,  reasonable  to  infer  that  better 
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factor 
NC,NIC 
NC,SMTCA 

NIC,SMTCA 


ALGOL-like  language 
AN         AE 
.5^3        .662 
-.i+2U      -.^75 
-.U08      -.U86 


KAIL-like  language 

AN  AE 

.jQk  .762 

-.3^5  -.381 

-.510  -.593 


Abbreviations 
AN 
AE 
NC 
NIC 
SMTCA 


Automorphic  Numbers  Problem 

Arithmetic  Expression  Problem 

Number  Correct 

Number  Initially  Correct 

Standardized  Mean  Time  to  a  Correct  Answer 


Table  h.      Correlations  with  Correct  Answer  Statistics 
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ALGOL-like  language 

.188 
-.083 

.233 

.005 
-.236 

.035 


Class  standard  score 

correlated  with: 
NC  on  AW 
NC  on  AE 
SMTCA  on  AN 
SMTCA  on  AE 
ISE  on  AN 
ISE  on  AE 
TL 
Abbreviations : 

ISE       Initial  Self-evaluation 

TL        Time  Learning 

See  Table  k   for  other  abbreviations 

Table  5.   Data  Correlated  with  Class  Standard  Score 


KAIL-like  language 

•  503 

.327 

-.1+05 

.0U9 

.35^ 

.110 

-.61+8 
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students  were  able  to  grasp  the  newer  KAIL-like  constructs  more  easily  than 
poorer  students.   The  selector  was  initially  unfamiliar;  the  more  intelligent 
students  understood  the  construct  quicker  and  therefore  performed  better.   All 
subjects  were  reasonably  familiar  with  the  ALGOL-like  language  constructs 
before  the  experiment  began.   Since  most  of  the  questions  were  relatively 
simple,  poorer  students,  drawing  from  their  background,  were  able  to  perform 
as  well  as  better  students. 
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5 .      CONCLUSION 

5.1  Summary  of  Results 

Subjects  using  the  KAIL-like  language  answered  more  questions  correctly 
than  subjects  using  the  ALGOL-like  language.   Disregarding  order,  they  also 
though  that  they  initially  better  understood  the  arithmetic  expression  program. 
These  results  were  significant. 

Of  the  four  groups,  the  group  that  performed  best  used  the  KAIL-like 
language  and  took  the  arithmetic  expression  problem  last.   This  group  worked 
the  program  which  maximized  the  use  of  the  selector  after  having  become  familiar 
with  the  KAIL-like  language.   It  was  hypothesized  that  these  subjects  would 
perform  best;  and  they  did. 

Statistics  on  the  number  of  questions  initially  answered  correctly, 
average  time  taken  to  obtain  a  correct  answer,  and  initial  and  final  self- 
evaluations  all  exhibited  trends  toward  the  KAIL-like  language.   No  performance 
statistics  favored  the  ALGOL-like  language. 

The  results  supported  the  basic  hypothesis.   Programmers  read  and 
understood  the  KAIL  selector  more  easily  than  an  equivalent  set  of  traditional 
language  constructs. 

5.2  Suggestions  for  Further  Research 

The  results  indicate  that  further  research  is  likely  to  be  fruitful. 
This  experiment  tested  for  the  subject's  ability  to  read,  understand,  and  to 
some  extent  to  learn  the  syntax  and  semantics  of  the  KAIL  selector.   Other 
experiments  should  be  conducted  to  test  a  programmer's  ability  to  write, 
debug,  and  modify. 

Since  the  experiment  used  CS  201  students  as  subjects,  the  results  are 
valid  for  this  category;  but  other  programmers  may  perform  differently.  Experi- 
ments using  a  wide  variety  of  subjects  are  likely  to  be  fruitful. 
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In  order  to  administer  this  experiment,  some  automated  methodologies 
for  psychological  experiments  in  computer  programming  were  developed.   These 
should  be  refined,  and  others  should  be  developed.   Because  more  and  better 
data  can  be  gathered  and  because  computer  resources  can  be  exploited,  automated 
methods  are  likely  to  be  useful  for  experiments  in  computer  programming. 
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APPENDIX  A 
FULL  GRAMMAR  OF  THE  KAIL  SELECTOR 


selector 


selection 
control  : 
type  : 
choices  : 
comparand 
action  : 


selection 

|  seltag  "_"  selection  "_"  seltag 

|  "*"  selection  "*" 

|  seltag  "_"  "*"  selection  "*"  "_"  seltag 

"["  statement-sequence  [control  choices]   "]" 

type  [expression] 

"if"  |  "while"  |  "until" 

A |  "|"  [comparand]  statement-sequence  [action]  choices 

[relation]  expression  ":"  |  " | "  "else"  ":" 


"exit"  [seltag]   "again"  [seltag] 
expression-selector  :  expression-selection 

seltag  "_"  expression-selection  "_"  seltag 


expression-selection  :   "["  statement-sequence  control  expression-choices  "]" 
expression-choices  :   "|"  [comparand]  expression 

"|"  [comparand]  expression  expression-choices 


Restrictions : 

1.  if.  may  only  *>e  used  with  [...], 

2.  while  and  until  may  only  be  used  with  *[. 


.]•• 


3.   seltags  must  match. 
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APPENDIX  B 
SUBJECT  BACKGROUND  DATA 

The  table  on  background  information  is  understood  as  follows: 


Group : 


language 


problem  order 


group  #1 
group  §2 
group  #3 
group  #U 


ALGOL-like  Automorphic  Numbers  problem  first 

KAIL-like  Automorphic  Numbers  problem  first 

ALGOL-like  Arithmetic  Expression  problem  first 

KAIL-like  Arithmetic  Expression  problem  first 


Education  Level: 

1.  Freshman 

2.  Sophomore 

3.  Junior 
h.  Senior 

5.  Four  years  plus,  but  no  degree 

6.  One  year  graduate  school 

7.  Two  years  graduate  school 

8.  Three  years  graduate  school 

9.  Four  or  more  years  graduate  school 

Language  Experience: 

1.  Never  heard  of  it 

2.  Knew  of  its  existence 

3.  Knew  a  little  about  it  or  had  written  a  program  in  it 

k.      Knew  it  fairly  well  or  had  written  several  programs  in  it 
5.   Knew  practically  everything  about  it  or  had  programmed 
extensively  using  it 
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PLATO  Experience: 


, _ _    subjects  with  PLATO  experience 

100  x  u - ; c 

number  in  group 


Class  Standard  Score  in  CS  201: 

mean:  500 

standard  deviation:   170 
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Group 


#1 

#2 

#3 

#h 

Number  in  Group 

20 

IT 

20 

15 

Education  Level 

2.8 

2.3 

3.1 

2.7 

Language  Experience 

2.2 

2.2 

2.1 

2.2 

ALGOL  68 

2.1 

1.8 

1.9 

1.8 

ALGOL  W 

1.5 

1.6 

1.8 

1.8 

COBOL 

2.3 

2.2 

2.3 

2,3 

EULER 

1.2 

1.2 

l.U 

1.0 

FORTRAN 

3.7 

3.8 

3.6 

3.7 

KAIL 

1.1 

1.2 

1.1 

1.0 

LISP 

1.1* 

1.2 

1.3 

1.1+ 

PL/1 

k.k 

h.5 

1+.2 

h.5 

SNOBOL 

1.6 

l.T 

1.8 

1.8 

TUTOR 

2.1+ 

2.6 

2.1 

2.2 

PLATO  Experience 

75$ 

652 

6o% 

872 

Class  Standard  Score 

U89 

517 

512 

513 
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APPENDIX  C 
PROBLEMS  AND  PROGRAM  SOLUTIONS 

Automorphic  Numbers 


Problem 


This  program  finds  all  automorphic  numbers  which  are 
less  than  or  equal  to  the  number  read  at  the  beginning  of 
the  program.  '  R  number,  n,  represented  by  djd?  ...  d^  is 

•i  Qo  ...  di. 


automorphic  if  the  last  k  digits  of  n  are  dido  ...  di, . 


Some  examples  of  automorphic  numbers  are 


Rs  you  can  see,  the  squares  of  these  numbers  end 
with  the  digits  which  represent  the  original  number. 
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ALGOL-like  Solution 

i  nt eger  i , j , k , n , i  sqr ; 
read  n; 

1  <?=  1  ;   k  <?  3; 
wh i 1 e  i  £  n  do 
i  sqr  <£  i  *  i ; 

j  *  i; 

wh  i  1  e  j  '  s"  l  do  j  <J=  j  *  1 J0 ' ;  od ; 

if  i=(isqr  -  j  *  (jsqr*j ) )  then  write  i,isqr;  fi; 

case  k 

0  i  <t=  i  +  4 ;   k  <J=  1 ; 

4  i  4=  i  +  1;   k  <!=  2; 


i    <>    i  + 


k  4=  0: 


od ; 


KAIL-like  Solution 


integer    i, j ,k,n, isqr; 

read   n ; 

l    <r    1  ;      k   <*=  fS ; 

*E    i    *    n    I 

l sqr   ♦    i  *  i ; 

j  *  i; 

>:<[  j  '<   i    8   j   <=  1*10;    1  ; 

[    i a  (isqr   -    j*(isqr*j))     I    write    i,isqr;     3; 
E   k 

L     i    <=    i    +    4;      k   <=    1 


<t=    i    +    1 ;       k.  <r 


i    ■$=    l    + 


0 


1. 


in 


Arithmetic  Expressions 


Problem 


This  program  analyzes  arithmetic  expressions  involving 
the  symbols  "+",  "-",-and  "a".   It  determines  whether  an 
expression  is  syntactically  correct,  deletes  unnecessary 
symbo 1 s ,  and  out  put  s  t  he  resu 1 1 . 


Thus,  for  example,  the  expression  "-+a-+-a+-a"  become* 
"-a+a-a",  and  the  string  "-a-"  is  in  error. 


To  make  the  program  easier,  the  input  is  encoded  as 

"  + "  =  l 

"  _  ••  _  ■-> 

"a"  =  3 
"he  last  input  is  always  a.  .9'  denoting  ^r\d   of  file  (eof)  . 


Thus ,  t  he  above  express  i  on  "  -  +  a  -  +  -  a  +  -  a "  is  encoded  f^>r 
l  nput  as  "213212-312  3  0 "  . 


k2 


ALGOL-like  Solution 

comment   0  =  eo  f ,  1  ■  "  + " ,  2  ■  " - " ,  3  =  " a " ; 

integer  i, state; 
char  act er  pr  f x ; 
state  4=  1 ; 
while  state  2:  0  do 

case   state  j 

•    i  ii   _  ii        ii ii        it  „  ii        ii  _  it        li__.ll. 

:e    write     e    ,    r    ,    r    ,    o    ,    r    ; 

state  <£  - 1 ; 
:  t    pr  f x  <J=  "  "  ;   read  i  ; 

state  <*=  case  l  :  e  0  :  t  1  :2  2  :3  3  esac; 


:  2  pr  1  x  <s= 

state  4=  case  i  :0  0  :x  2  :2  1  :3  3  esac; 
:  ri  write  prfx,"a";   read  i; 

state  ■>   case  i  :e  -1  :t  4  :2  5  :3  0  esac; 
;  ,    pr :  x  ■£   +  ;   read  l  ; 


state  <{=  case  i  :e  0  :x  4  : 
5  prfx  *  "-" ;   read"  i; 


state  v=  case  i  :  -  0  i4     5  :^  4  :  -  3  esac; 
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KAIL-like  Solution 


in-  a^  «  »  II  <~i  II  II  *\  II  II    . 

comment      0   ■   eox ,     I  ■       +    ,     2    =      -,3=      a; 

mtege"  l, state; 
character  prfx; 
state  ♦  1 ; 
*[  state 

le  write  "e\"r","r\"o","r"; 

state   <r   -1: 

i  j    pr  f x   <J=    "    "  ;  read    i ; 

state   *    [    i  |0    0    Ij     1    l2    2    l3    3    3; 

l2    prfx   <>    "-" ;  read    i; 

state  *    [    i  |e    0    Bj    2    l2    1    l9    3    3; 
S^    write   prfx,  "a.";       read    i; 

state   +    [    i  |f    -1    lt     4    l2    5    I,    ff    ]; 

1 4  pr  f x    j=    "  +  "  ;  read    l ; 

state  +    [    i  I,    0    I*    4    l3    5    |3    3    ]; 

15  prfy.   <>    "-";  read    i; 

state   *    [    i  Igilj    5    l2    4    l3i   3    ]; 
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APPENDIX  D 
QUESTIONS 
Automorphic  Numbers  Problem 

1.  Each  time  the  write  statement  is  executed,  how  many  values  are 
output  ? 

2.  How  many  times  does  k  appear  on  the  left  hand  side  of  "<="? 

3.  List  the  values  that  k  can  have  during  program  execution. 
(Separate  values  with  ","  or  space.) 

U.   How  many  times  does  i  appear  on  the  left  hand  side  of  "<="? 

5.  List,  in  order,  the  first  seven  values  that  i  can  have. 
(Separate  values  with  ","  or  space.) 

6.  List,  in  order,  the  first  four  values  that  isqr  can  have. 
(Separate  values  with  ","  or  space.) 

7.  If  i  is  1537  >  what  value  does  j  have  when  j  *(isqr  f  j)  is 
evaluated? 

8.  If  i  is  5>  what  is  (isqr  -  j*(isqr  -r  J))? 

9.  If  n  is  52,  how  many  times  is  the  statement  "j<=l"  executed? 
10.   Approximately  what  is  the  probability  that  i  is  equal  to  n  the 

last  time  through  the  outer  loop? 

Arithmetic  Expression  Problem 

1.  How  many  read  statements  are  in  the  program? 

2.  How  many  times  does  state  appear  on  the  left  hand  side  "<="? 

3.  In  how  many  different  statements  is  "-"  assigned  to  prfx? 
h.      In  how  many  different  statements  is  "+"  assigned  to  prfx? 

5.  If  the  input  string  consists  only  of  "0"  what  is  the  output? 

6.  If  the  input  string  is  "3  0" ,  what  is  the  output? 


U5 


7.  A  sequence  of  two  statements  can  be  simplified  to  "read  state". 
What  statement  immediately  precedes  these? 

8.  Which  statement  can  be  replaced  by  "state  <=  -13"  without 
altering  the  program  behavior? 

9.  When  the  input  is  "12  3  3  0" ,  what  sequence  of  characters  is 
output? 

10.   What  is  the  sequence  of  state  values  for  the  input  "2313  0"? 
(Separate  values  with  ","  or  space.) 
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APPENDIX  E 
PERFORMANCE  STATISTICS 

A  three-way  analysis  of  variance  using  the  approximate  method  of 
unweighted  means  was  used  to  compute  these  statistics.  The  three  factors 
were  language,  order,  and  problem.  Subjects  took  both  problems,  but  only 
in  one  order  using  one  language. 


UT 


Number  of  questions  answered  correctly 


Source  of  variance 


df 


MS 


F   significance 


language 

order 

language  x  order 

subjects  within  levels  of 

language  and  order  (error) 
problem 

language  x  problem 
problem  x  order 
language  x  problem  x  order 
subjects  within  levels  of  language 

and  order  x  problem  (error) 


1  1T.2U  5.3U  <2.5/& 

1  6.U9  2.01 

1  3. 97  1.23 

5U  3.22 

1  Ik. 63  13.55  <1% 

1  2.U5  2.27 

1  0.08  0.08 

1  2.62  2.1+3 


5h 


1.08 


1+8 


Number  of  questions  answered  correctly  on  first  attempt 


Sources  of  varience 


df 


MS 


F     significance 


language 

1 

2.40 

0.1+8 

order 

1 

5-91 

1.19 

language  x  order 

1 

6.05 

1.22 

subjects  within  levels  of  language 

and  order  (error) 

5^ 

h.97 

problem 

language  x  problem 
problem  x  order 
language  x  problem  x  order 
subjects  within  levels  of  language 
and  order  x  problem  (error) 


1 
1 
1 
1 

5^ 


10.68  5.61+ 

O.n  0.06 

0.3I+  0.18 

0.10  0.05 

1.89 


<2. 
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Average  time  taken  to  answer  a  question  correctly 

To  obtain  these  statistics,  raw  data  on  the  time  taken  to  answer  a 
question  correctly  was  first  transformed  to  N(0,l),  and  an  average  for 
each  subject  on  each  problem  was  computed.   Data  on  questions  answered 
incorrectly  was  ignored. 

Sources  of  variance  df      MS       F     significance 


language 

1 

0.1+7 

1.3»i 

order 

1 

0.U9 

1.39 

language  x  order 

1 

0.06 

0.17 

subjects  within  levels  of 

language 

\ 

and  order  (error) 

5* 

0.35 

problem 

1 

0.00 

0.00 

language  x  problem 

1 

0.03 

0.18 

problem  x  order 

1 

0.31 

1.9^ 

language  x  problem  x  order 

1 

0.13 

0.82 

subjects  within  levels  of  language 

and  order  x  problem  (error)  5^     0.l6 
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Initial  self-evaluation 


Sources  of  variance 


df 


MS 


significance 


language 

order 

language  x  order 

subjects  within  levels  of  language 

and  order  (error) 
problem 

language  x  problem 
problem  x  order 
language  x  problem  x  order 
subjects  within  levels  of  language 

and  order  x  problem  (error) 


1  1.57  0.31 

1  2.12  O.Ul 

1  3.19  0.62 

5^  5.13 

1  15. Qh  7.98 

1  U.93  2.U8 

1  1.30  O.65 

1  0.39  0.19 


<1% 


5^ 


1.99 


Final  self-evaluation 
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Sources  of  variance 


df 


MS 


significance 


language 

order 

language  x  order 

subjects  within  levels  of  language 

and  order  (error) 
problem 

language  x  problems 
problem  x  order 
language  x  problem  x  order 
subjects  within  levels  of  language 

and  order  x  problem  (error) 


1 

l.hl 

0.18 

1 

2k.  61 

3.02 

1 

0.81 

0.10 

5h  8.15 

1  7>+.29  31.1+3 

1  0.33  o.lI+ 

1  0.08  o.03 

1  0.69  0.29 


5h 


0.2U 
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Difference  "between  initial  and  final  self -evaluation 


Sources  of  variance 


df 


MS 


significance 


language 

order 

language  x  order 

subjects  within  levels  of  language 

and  order  (error) 
problem 

language  x  problem 
problem  x  order 
language  x  problem  x  order 
subjects  iwthin  levels  of  language 

and  order  x  problem  (error) 


1 

0.00 

0.00 

1 

12.30 

i.6o 

1 

0.78 

0.10 

5h  1.66 

1  21.52  7. 01| 

1  2.72  0.89 

1  0.75  0.2U 

1  o-o^  0.01 


<  2. 


5^ 


3.05 
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APPENDIX  F 
CORRELATIONS 

A  missing  data  correlation  program  was  used  to  compute  these  statistics 
In  the  following: 

AN  =  Automorphic  Numbers  Problem, 

AE  =  Arithmetic  Expression  Problem, 

*  =  significant  with  probability  .010, 

**  =  significant  with  probability  .005. 
Correlation  factors : 

1.  Number  correct  (AN) 

2.  Number  correct  (AE) 

3.  Number  initially  correct  (AN) 
h.      Number  initially  correct  (AE) 

5.  Standardized  mean  time  to  a  correct  answer  (AN) 

6.  Standardized  mean  time  to  a  correct  answer  (AE) 

7.  Initial  self-evaluation  (AN) 

8.  Initial  self-evaluation  (AE) 

9.  Final  self -evaluation  (AN) 

10.  Final  self -evaluation  (AE) 

11.  Difference  in  self -evaluations  (AN) 

12.  Difference  in  self-evaluations  (AE) 

13.  Language  knowledge 
ik.  Class  standard  score 
15.  Time  spent  learning 
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